Method and system for processing messages within the framework of an integrated message system

ABSTRACT

A method and system for processing messages within the framework of an integrated message system. Recipients of messages in an integrated messaging system are provided with an authentic impression of the received message. In a first step, a message received within the framework of an integrated messaging system is automatically translated. Language detection and dictation system is provided. The message contents of the incoming message as well as its segments and parameters are simultaneously utilized to generate additional information regarding the sender and the information, which is suitable to give the recipient an impression of the received message in the most authentic form possible.

FIELD OF THE INVENTION

The present invention relates to an integrated messaging system and to amethod and a system by which messages that are transmitted within theframework of an integrated messaging system, such as a mailbox, areoffered to a recipient with an information content that is not readilyapparent from the unprocessed message.

BACKGROUND INFORMATION

Messaging systems (Mobilbox, voice mail systems) are available in themarketplace (e.g., http://www.t-mobile.de/mobilbox). Depending on thesystem structure, voice messages are often stored here as attachments oftext messages (e-mail attachments together with sender information (suchas sender identification (e.g., CLI, etc.)), arrival time, etc.

There are also various speech recognition systems in the telephonesector (e.g., http://www.nuance.com; http://www.scansoft.com, etc.), andsystems for desktop dictation recognition (e.g., Dragon NaturallySpeaking at http://www.scansoft.com, and IBM ViaVoice athttp://www.ibm.com/software/voice/viavoice), which are able to convertspoken language into text information. These systems have at theirdisposal deterministic grammars (e.g., nuance grammar specificationlanguage) or grammars based on internal statistics (e.g., Scansoft). Thelatter are produced mostly in an application-based or domain-basedmanner with the aid of a large quantity of textual data, for example,from newspaper articles, books or language data compilations, so as tocalculate detector-internal probability models for word transitions fromword sequences occurring in these texts.

To assist in multilingual speech recognition, newer speech-recognitionsystems also offer internal functions for dynamic foreign languagedetection (e.g.,http://www.nuance.com/assets/pdf/nuance85_datasheet_(—)020304.pdf). Aforeign language detection based on algorithms from speech-signalprocessing may be implemented by, for example, analyzing the frequencyat which phonemes occur in a spoken utterance, which are provided by aphoneme-based speech detector.

Reference WO 03/024073 purportedly refers to a method for presentinginformation from telephone messages to a user where a search forpredefined information is implemented within voice messages. The methodapparently includes the steps of receiving incoming telephone messagesand detecting language in the incoming telephone messages by searchingthe incoming telephone messages for at least one previously definedinformation category. If a previously predefined information category isfound in the detected language, the information is reproduced for theuser. And, single-language systems are not always able to be used if,for instance, messages in different languages are involved. However,this and similar methods fail to take into account that the receivedvoice messages may have been recorded in a language other than the onerequired for the recipient.

Also available are systems for translating written language and spokenlanguage (e.g., Fu-Hua Liu et al. “Noise robustness in speech to speechtranslation”, IBM Tech Report RC22874, 2003), in which a method forreducing noise on the speech detection level is described.

Available translation systems (e.g.,http://penance.is.cs.cmu.edu/11-733/Slides/JoyZhang.pdf andhttp://www.linguatec.de/products/pt2004/index.shtml), assume knowledgeof the source and target languages for selection of the appropriatetranslation module.

Available approaches appear to provide that no automatic foreignlanguage detection takes place where the language of the incomingmessage is detected, and specifically on the source language side.

Also available is classifying messages both according to acquiredsupplementary data and according to their contents, and to categorizethem in accordance with the mailbox owner's or the system operator'sintentions. Reference European patent no. 1298872 purportedly refers toa method for processing messages in a unified messaging system wheredifferent message categories are defined so that each message in aunified messaging system is able to be assigned to a rule that is partof at least one category. For example, the message category may includecategories that are assigned to specific types or formats, as well ascategories that are freely definable. It is possible to define specificrules for the message categories which are used for message assignment.This approach, too, fails to take a possible multi-linguality ofincoming messages into account.

Reference US 2002/0069048 presents some general ideas for translatingmessages with output in an audio format, without a detection of thesource language having been described in greater detail. Such approacheshave the disadvantage that during the dialogue with the system themessage recipient is obliged to explain in cumbersome detail in whattarget languages he wishes the audio output of each individual messageto occur.

Also available are approaches that are based on detecting various sensordata and their transmission via a telecommunication network. ReferenceDE 101 49 049 A1, for example, purportedly refers to a method and asystem for creating and modifying a virtual biological representation ofthe users of computer applications based on biological parameters of theuser of the computer applications.

SUMMARY OF THE INVENTION

Exemplary embodiments and/or methods of the present invention providerecipients of messages in an integrated messaging system with the mostauthentic impression of the message possible. This may apply both to themessage itself, which is to be offered to the user in comprehensibleform, and to additional information in connection with the message andmessage sender that cannot be gathered explicitly from the messageitself. Exemplary embodiments and/or methods also may solve the specificproblem that the incoming message is a message expressed in a languagethat is unknown to the recipient. Exemplary embodiments and/or methodsmay improve the recipient's understanding of the message content byoffering the message as an event with the highest possible authenticityand an increased information content. At the same time, the solution isaimed at improving the structuring of message groups and at improvingthe navigation within the messages itself.

Exemplary embodiments and/or methods of the present inventions offer themessages to the recipient in the most authentic manner possible and as areal experience, engaging as many of the recipient's senses as possible.This may increase the performance especially for a group of persons thatis active on an international level for the most part. This may applyspecifically to persons working in international management or in areasof international research, development and culture.

In the following discussion, a message or a message document denotes adocument and/or a document segment received via a messaging system,which may include a variety of information formats such as, for example,text, video/image, audio/speech, biometrical information, smellinformation, gesture information, temperature information, etc. Both theattributes of the message document such as receiving time, sender'saddress, sender identification, sender's name, sending time etc., aswell as appendices and references may be considered part or segment ofthe message document.

The term message segment is understood to denote a part or aninformation unit of a message document such as a sentence, a sentencesequence, a word, a word sequence, a bit sequence, a paragraph, aparagraph sequence, or an attachment. Message segments may have anidentical and/or different format, for instance an audio format, a textformat, a video format, an image format, or also other possible formats.A link is possible as well.

An identification within the meaning of the present invention may bemade up of one and/or a plurality of classes.

In exemplary embodiments and/or methods of the present invention, theindividual classes can be technical features for, e.g.,

1. the customer identification such as

-   -   customer number    -   PAN,    -   a personal identification card number;        2. the communication identification such as    -   CLI (calling line identification)    -   HLR (home location register)    -   IP address (Internet protocol)    -   call number, telephone number    -   telecard number;        3. the device identification such as    -   IMEI (international mobile equipment identity)    -   Telecard number    -   SIM card (subscriber identity module) smart card;        4. the transaction identification for the communication        transaction;        5. biometrical identification such as    -   fingerprint    -   voice print    -   iris.

Exemplary embodiments and/or methods of the present invention mayprovide that, in a first step, messages arriving via an integratedmessaging system are translated into a language specified or preset bythe recipient. To this end, incoming voice messages, for example, afterconversion into source-text information by a speech recognition anddictation system, may be automatically translated into a desired targetlanguage and then made available to the user.

At the same time, in further embodiments and/or methods, the informationcontents of the incoming message and its segments and parameters isutilized to generate additional information regarding the sender andregarding other message-related facts that are suitable to convey to therecipient an understanding of the creation of the message in the mostauthentic form possible. In the process, information about the senderthat may possibly not be available in system-internal data records, ismeant to be generated as well.

In exemplary embodiments and/or methods, storing and/or collectingcorresponding information such as, for example, the mailbox owner'spreferred target language, in an owner data area 170. The owner dataarea is a memory area for acquired and/or ascertained owner data such asthe ID code, user identification, mother tongue, desired targetlanguage, preferred TTS voice, preferred playback speed, target callnumbers and search keywords for important messages. The setting may bemade via a Web interface and browser, for example, by voice input in amulti-modal or uni-modal manner, or by e-mail via mailbox 30 of themailbox owner. Once the identification has been detected, embodimentsand/or methods of the present invention provides an additionallanguage-evaluation module 100 which includes an algorithm for selectingthe most likely language. Language-evaluation module 100, utilizing allavailable information, is used for the ultimate determination of thesource language of an incoming message. Additional language-evaluationmodule 100 is required since the use of automatic methods based onlanguage detection by the application of methods of language-signalprocessing or methods that are based on language detection on the basisof a text, may lead to false results. False results in the above sensemay occur, for example, if a German message is spoken by a speakerinexperienced in German phonetics, or in texts with mixed languages. Viaa sender data area for acquired and/or ascertained sender data 60, it isalso possible to prepare a profile for the preferred sender language forsenders often encountered in the mailbox, such a profile being settableboth administratively and after analyzing already existing messages ofthe particular sender in the sender data area for acquired and/orascertained sender data 60. In further embodiments and/or methods of thepresent invention, this feature may need to be combinable with anotherfeature of the present invention since different languages may be usedin different situations, for example, when messages from a sender areaddressed to one recipient in the German language and to additionalrecipients in the English language. Similar also may apply if only anidentification were analyzed, for example, a CLI with an internationalprefix. For these reasons, the sender data area for acquired and/orascertained sender data 60 may be configured to store not only theparameters explicitly entered by the mailbox owner but also parametersthat the system ascertains from the incoming messages of the sender.This can be, among other things, the most likely language spoken, theascertained gender of the message sender, the ascertained emotionaldisposition of the message sender and/or the age category of the messagesender. In addition, the mailbox owner may have the explicit possibilityof overwriting data stored in the sender data area for acquired and/orascertained sender data 60, for example, via Web interface and browser,or over the telephone, in a multi-modal or uni-modal manner, forinstance by speech recognition. In further embodiments and/or methods,an additional development of the memory area for acquired and/orascertained sender data 60 is to retain or store the history ofascertained sender data for later analysis. For example, if a mailboxowner always deletes the messages from a sender without listening tothem or reading them, starting with a number predefined in the system,it may be possible to generate a parameter linkage within the systemthat classifies the messages from this sender as spam and either deletesthem immediately or shifts them into a memory area having an extremelylow priority. As a result, such messages are always played last.

The linking of messages by an individual receiver mailbox itself or byits sender CLI, as well as other sorting criteria such as the receivingtime, are available in the related art. Exemplary embodiments and/ormethods of the present invention now allow a linking of messagesaccording to their content in a manner that goes beyond the individualmailbox, if appropriate. A considerably simplified classification ofmessages results from the expansion of the sender addresses/senderinformation (CLI, HLR, e-mail address, time information) by brief,abstracting content descriptions such as class names.

Systems for acquiring various sensor data and their transmission via atelecommunication network are available. Supplementing messages withinthe messaging system by these data allows both more refined searchcriteria within individual message classes and the creation of novelmessage classes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an embodiment of the present invention.

FIG. 2 shows an architecture for an embodiment of an exemplarydevelopment of classification block 500.

DETAILED DESCRIPTION OF THE DRAWINGS

The exemplary embodiment that is described with the aid of FIGS. 1 and 2requires, for example, an integrated messaging system designed as amailbox system. However, another possibility is using the solution in anonline application. Hereinafter, the solution will be described inconnection with mailbox 30 of a mailbox owner.

In addition to the messages that are intended for the mailbox owner andresult from the memory area for incoming text messages and supplementarymessages 40, as well as from the memory area for incoming voice messagesand supplementary messages 50, mailbox 30 may be aware also of data ofthe owner that are stored in the owner data area for acquired and/orascertained owner data 170, and also of data of the message sender suchas the owner CLI. Supplementary messages are to be understood assupplementary information such as additional text, image and videoinformation as well as information of other media types (alsomedia-spanning types). The same also applies to data relating to thesender of the message, which had been compiled previously already andwere transmitted via communication network 20.

Mailbox 30 is assigned at least one ASR module 70 with algorithms forlanguage detection, a module for language detection 80 for foreignlanguage detection, and at least one language-evaluation module 100 withan algorithm for selecting the most likely source language of a messagereceived from a sender 10. The language-evaluation module 100 weightsthe results of the intermediate results of the language detection bymodule for language detection 80 and possibly at least one module forlanguage recognition/language detection on the basis of a text 90,possibly by the country prefix from the sender identification and thepreferred language that may have been entered in the sender data areafor acquired and/or ascertained sender data 60, providing as a resultthe most likely source language by applying suitable, possiblystatistics-based algorithms.

The result may be used not only for the further correction of thepreferred language of the sender in the data record assigned to thesender's transmitted identification, which is stored in the sender dataarea for acquired and/or ascertained sender data 60, but also for theautomatic selection of the source language by translation system 190.For example, the emotional disposition, the preferred language, theprobable gender, age or age group of the sender may be recorded in thesender data area for acquired and/or ascertained sender data 60. Thetarget language of the automatic translation is selected on the basis ofthe data of the individual mailbox owner, which is stored in the ownerdata area for acquired and/or ascertained owner data 170.

Translation system 190 may be made up of a large number ofproducts/translation systems of different manufacturers. An automaticselection of the language combination, and thus the required translationmodule, takes place by the implemented source and target languagedetermination. If a directly required combination is not available, itis automatically searched for a possible interlingua combination. In thefollowing, exemplary translation system 190 is described in greaterdetail.

To determine the source language, which is accomplished via languageevaluation module 100, an additional algorithm for the selection of themost probable language is required in which the input information may beweighted differently, for instance, either in general or as a functionof the message type, and/or statistical methods are applied. Here, thealgorithm always determines the language that delivers a maximum valuefor a predefined function for the given message or for the segment of amessage at the given time.

The language of a message and/or the segments of a message may beascertained in the following manner, for example:

L=argmax(W _(CLI)(Lx)+W _(LD) *C _(LD)(L _(x))+W _(T) *C _(T)(L _(x))+W_(H) *N _(H)(L _(x))+ . . . +W _(n) *C _(n)(L _(x))),

with the following explanations:

-   L—determined language of a message and/or a message segment-   W_(CLI)—weighting factor for the country code of a message (such as    +49 for Germany) in combination with a specific language Lx-   W_(LD)—weighting factor for the language detection by application of    speech-signal processing methods (80)-   W_(T)—weighting factor for language detection on the basis of a text    (90)-   W_(H)—weighting factor for the languages determined from previous    messages of a sender-   W_(n)—weighting factor for additional input parameters of (100),    among them image and video, for instance-   L_(x)—concrete argument (concrete language from a list n of possible    languages) for which a functional value is calculated from n    functional values-   C_(LD)—confidence value ascertained by 80 (such as probability) of    the language determined by 80-   C_(T)—confidence value determined by 90 (such as probability) of the    language ascertained by 90-   N_(H) Number of values for L_(X) previously ascertained for a sender    number of the language values previously ascertained for a sender-   C_(n)—confidence value (such as probability) of source language Lx    ascertained by additional methods.

The weighting coefficients may be given individual pre-adjustment valueseither by the system administrator and/or the mailbox owner. However,the preadjustment values may also be determined in an automated manner,for instance within the framework of a neural network. The confidencevalues for the given message with regard to the languages supported bythe system are ascertained in an ASR module 70, which may be configuredas server or server group with algorithms for language detection, and/orthey are ascertained by a module for language detection 90. In themodule for language detection 90, the language detection is implementedon the basis of a text. If no confidence value is determined for asupported language, the value is assumed to be 0 (zero), for example. Ifonly the language is determined as the result of a module, without theprovision of a confidence value, some other fixed value such as 1 (one)may be assumed as confidence. The mentioned function is implemented foreach source language supported by the system. The particular languagefor which the maximum value is achieved according to the previouslymentioned algorithm will be considered the source language. If the samemaximum value or maximum values that are very similar is/are reached fora plurality of languages, the message cannot be translated. In thiscase, the recipient (mailbox owner), when reading/playing back themessage, may perhaps choose between the languages—now restricted intheir number—, or a system administrator and/or the mailbox ownerreceive a message, for example, with the request to correct theweighting coefficients of the algorithm.

The result of the language detection of the source language is used bothfor the further correction of the preferred language of the sender bythe data from the sender data area for acquired and/or detected senderdata 60, and also for the automatic selection of the source language fortranslation system 190. Via the output module for target languages 140,which is designed as software interface with target languages S₁-Sm, inconjunction with the owner data area for acquired and/or detected ownerdata 170, the preferred language of the mailbox owner, and thus thetarget language into which the message is to be translated, isascertained.

The translation module with decision matrix 130 is connected both withat least one input module for source languages 110 having sourcelanguages S1 to Sn, and with at least one output module for targetlanguages having target languages S1 to Sm. The decision matrix oftranslation module 130 also may have an n-dimensional design. Thetranslation module with decision matrix 130, which encompasses differenttranslation modules from a first language into a second language, iscontrolled by the input module for source languages 110 and by themodule for selection of the target languages 120, which is configured assoftware interface in translation system 190. The control of theindividual translation modules of the translation module with decisionmatrix 130 is implemented via the decision matrix of translation module130 into which both the result of the detection of the source languageand the ascertained target language are input automatically.

The translation module with decision matrix 130 may be composed ofproducts or translation systems of different manufacturers. Due to thesource and target languages being determined according to theaforementioned principles, an automatic selection of the languagecombination(s) takes place, and, thus, also a selection of thetranslation module required for the translation. For instance, if arequired translation module for translation from a language S1 into alanguage S3 is not available, but translation modules from language S1into a language S4 and from language S4 into language S3 are available,the translation request is able to be carried out by combining thementioned translation modules and applying language S4 as interlingua. Aresulting lower translation quality also may be acceptable.

In addition to the data that result from the translation of the message,the message recipient may be given additional information about themessage by employing associative knowledge management systems, forexample, utilizing a large number of available data as well as dataobtained via classification and knowledge management and their linkingin integrated messaging systems. Knowledge management systems may beunderstood as, for example, search engines as well as associativeknowledge management systems.

In addition to the sender profile such as name, first name, senderidentification (for instance, CLI, personal ID . . . ), e-mail address,country code, gender, different call numbers, roll assignment for aparticular call number (such as consumer, business), the data record ofthe mailbox owner also notes which source languages may be excluded froma translation. If there is a match between the source and targetlanguage, or if a particular source language has been excluded fromtranslation in the data record of the mailbox owner, the utilization ofthat particular translation module of translation system 190 is excludedafter evaluation of the most probable source language and the owner dataof the data record of the mailbox owner. If the original message is avoicemail, the text body of the e-mail, ascertained via ASR module 70,is recorded in the memory area for incoming voice messages andsupplementary messages 50, and/or in a supplementary message as text inthe original language and, if available, as language text 50 atranslated into the target language, in the text body as well assupplementary data record in the memory area for supplementaryinformation of individual messages 52. If the original message was atext message, perhaps including an unspecified attachment, the textmessage is supplemented by supplementary text 40 a translated into thetarget language, if available, or the text in the target language willbe included in the supplementary data record. Each of the individualtext portions in different languages is supplemented by a supplementarydatum that characterizes the selected language. This allows selection ofthe correct pronunciation dictionary for the individual language in asubsequent multilingual reproduction of the texts of this message viaspeech synthesis.

In general or after successful search of key words stored in the ownerdata area for acquired and/or ascertained owner data 170, the messages,which have been converted into text, are able to be sent to the targetcall numbers correspondingly marked in the owner data area for acquiredand/or ascertained owner data 170, for example, as SMS, MMS. This searchfor specific characteristics may be utilized to filter out undesiredvoice messages (e.g., voice spam).

Corresponding internal deletion parameters and parameter linkages thatmake further analysis of the contents unnecessary may be generated onthe basis of the results. If the operator of the integrated messagingsystem detects the generation of such parameters and/or parameterlinkages in a multitude of its mailbox owners, it may use these togenerate universally valid parameters and/or parameter linkages in thedata memory with sample data 260. The data memory with sample data 260includes sample data and/or dynamic and/or static parameters orparameter linkages.

According to the present invention, the system shown in FIG. 1 also maybe utilized to record probable emotional states and/or probable agegroups and/or the gender of the sender of a message.

In addition to the already described embodiments/methods, thecorresponding system expansions also may include at least one module forspeaker classification 200 and/or at least one module for analyzingspeaker emotions 210 by prosody analysis and/or at least one module foranalyzing sender emotions 220 by semantic analysis of the messagecontents and/or analysis of the image message, at least one data memoryfor information regarding the languages that are able to be processed bythe available emotion-analysis module and/or translation module(s) 250,and/or, at least one data memory with sample data 260.

To ascertain the most probable emotional state of the sender of voicemessages via a module for ascertaining the most probable emotional state230, both prosodic information resulting from voice messages and resultsfrom the semantic analysis of available texts and/or additional mediatypes relating to the sender, such as image or video information, areevaluated. The gender and/or age of the sender of the message are/isdetermined via at least one module for speaker classification 200. Asadditional basis of a system for ascertaining the most probableemotional state of a caller—such system becoming more refined in thecourse of usage—corresponding statistical data of the caller, which arestored in the sender data area for acquired and/or ascertained senderdata 60, are analyzed. This is data from earlier analyses, which mayinclude, for example, weightings for individual events, similar to thedetermination of the most probable language. In addition tostatistically dynamically acquired data (such as owner behavior withspecific e-mail senders), the sender data area for acquired and/orascertained sender data 60 also includes the sender profile, forinstance the name, first name, sender identification (such as CLI,personal ID, etc.), e-mail address, country code, gender, different callnumbers, role assignment for a specific call number (such as consumer,business). Selected profile data are also able to be generated by thesystem and/or be provided with supplementary information (for example,probability values, confidences).

For example, if the mailbox owner has not created an entry for thegender of a sender, it is also possible to dynamically enter this valueagain in the sender data area for acquired and/or ascertained senderdata 60 once these data have been determined from the module for speakerclassification 200 and/or from the data of the module for ascertainingthe most probable gender 280, taking past results from the sender dataarea for acquired and/or ascertained sender data 60 into accountincluding supplementary data. This applies in a similar manner to theage group determination and to the analysis of emotions via the modulefor prosodic analysis of speaker emotions 210 and the module forsemantic analysis of sender emotions 220, respectively.

Since both the module for analysis of speaker emotions 210 and themodule for analysis of sender emotions 220 as well as the module forspeaker classification 200 may depend on the language used, the sourcelanguage determined via decision module 180 may be used to select thedata records of modules 200, 210 and 220 that are available for thislanguage. If appropriate, decision module 180 includes source-textforwarding to mailbox 30 for voice messages and/or further analysis. Ifthe source language is identical to the target language, no translationwill be provided. Otherwise, the source text will be forwarded totranslation system 190, and the information concerning the sourcelanguage be transmitted to the interface module for selecting the sourcelanguage 150.

If no analysis module is available for the selected source language,translation system 190 begins a search for translation modules thatprovide as result a language that is suitable for the semantic textualemotion analysis.

The ascertained probable emotional state of the message sender togetherwith the possibly ascertained information regarding age group and/orgender and/or time information is recorded in the sender's data recordfor further statistical analysis. Universally valid parameters and/orparameter linkages for analyzing sender information are specified bothstatically, by the system administrator, and are ascertained byanalyzing a multitude of parameter sets of the mailboxes of othermailbox owners 270 in the integrated messaging system. The desired datamay be accessed via Internet/Intranet 21.

Such parameter linkages may be used, for example, to determine thegender from the speaker classification and first name of the sender (ifavailable). Such parameter linkages may read, for example:

IF first name=“Andrea” AND international prefix=“+49” (Germany) AND(speaker classification=male; confidence <90%) THEN gender:=“female”; orIF first name=“Andrea” AND international prefix=“+39” (Italy) AND(speaker classification=male; confidence >60%) THEN gender:=“male”; orIF first name=“Andrea” AND international prefix=“+39” AND (speakerclassification=male; confidence=50%) THEN gender:=“undetermined”.

Instead of the international prefix it is also possible to use someother identification in this context. A universally valid parameter thusis valid for all mailboxes 30, and is used for messages of a specificowner. The aforedescribed parameters and/or parameter linkages, whichare able to be entered permanently by the administrator of the system orby the mailbox owner, are also referred to as static parameters. Dynamicparameters are generated by the system itself. Such a generation ofparameter linkages might be carried out in the following manner, forexample:

Upon detecting a larger number of senders having the first name “feta”with the international prefix +kk in more than N mailboxes of differentowners, and a gender detection from the speaker classification of“male”, confidence >90%, in more than M cases, the following dynamicparameter linkage, for instance, will be deleted

IF first name=“Peta” AND international prefix=“+kk” AND (speakerclassification=male; confidence=50%) THEN gender:=“female”,the following new parameter linkage being generated:IF first name=“Peta” AND international prefix=“+kk” AND (speakerclassification=male; confidence>=50%) THEN gender:=“male”.

The examples assume that the general probability for gender assignmentis approximately 50%, i.e., IF (speaker classification=male;confidence>=50%) THEN gender:=“male”; and IF (speakerclassification=male; confidence<50%) THEN gender:=“female”.

The confidence values may be scaled differently in different systems.

In addition to general parameters, exemplary embodiments and/or methodsfurther may involve defining and/or generating mailbox-internal, staticand dynamic parameters and their linkages. These parameters aregenerated solely on the basis of mailbox-internal data or entered by theowner, and they apply only to the messages of the mailbox owner, itbeing possible in some cases that external, universally valid parameterlinkages are overwritten by internal parameter linkages. Here, too, theprioritization of messages of specific senders, for instance, may bedefined, for example:

IF (message having sender identification X) AND (listening to messageshaving sender identification X broken off with 80% probability) THEN setpriority(X):=priority(X)−1;WHEN priority(X)<=0 THEN (skip playing of the message) etc.

More recent events may receive greater weight when ascertaining thebreak-off probability, for example, so that the parameter linkages areable to be rapidly adapted to current requirements.

As already described, there are dynamic and static parameters. Inaddition, default parameters are set when initializing the system in thefirst step. These default parameters may be overwritten by dynamic,system-generated parameters. These in turn are able to be overwritten bystatic parameters that result from permanent inputs of the mailbox owneror the system administrator.

Since both the language detection and the translation may require ageneral analysis of the semantic information, these operations may beprocessed in an overall module made up of ASR module 70 and the modulefor analyzing sender emotions 220, or also in separate modules fromdifferent manufacturers.

The sender data area for acquired and/or ascertained sender data 60,which is configured as database, forms the basis of a self-learningsystem for perfecting the determination of the source language, thesender's gender, sender's age group and the probable emotional state ofa message sender, utilizing parameter linkages that are modifiableinternally and externally.

The probable emotional state may also be used to prioritize messages.Furthermore, in this context, it is also possible to derive individualweighting factors of individual emotional states from frequent emotionalstates of particular senders, these weighting factors influencing theparameter linkages that are stored in the data memory having sample data260 and in the module for determining the most probable gender 280,which are analyzed when determining the most probable emotional state.

Ascertained sender data may be utilized for parameter control, forexample, to select the voice in the synthetic reproduction of textmessages (speech synthesis, text-to-speech).

Since the sender data ascertained from the individual messages need notbe complete, the system is able to obtain an interface that offers thepossibility of searching, for example, in company-internal addressdatabases, for further data that have not been taken into account yet.The available data are utilized as search parameters. Furthermore, thesystem must be robust with respect to faults, and error-tolerant. It mayhappen, for example, that the same sender is administered multiple timesin the system if, for example, he has left only a voicemail and ane-mail without leaving the call number. As soon as a mailbox owner, forexample, establishes a connection between the two sender datums via anadministration interface in that he amends the e-mail if the CLI isgiven, and/or the system amends the corresponding parameter by automaticevaluation of the message contents in one case, all messages from thissender bearing the sender's CLI or e-mail address are able to beassigned to a sender data record. This may be done not only for anindividual mailbox but for all sender data records of the sender dataarea for acquired and/or ascertained sender data (60) of the system'smailboxes.

Furthermore, in further embodiments/methods, already acquired orgenerated supplementary data may be classified according to theircontent. The classification may occur either according to thespecifications of the mailbox owner and/or according to thespecifications of the system operator. The generated supplementary datamay be combined into message groups (clusters) which are able to beprocessed jointly. Message groups may be combined according to thepredefined priorities, for instance according to sending date, senderidentification (such as CLI, sender ID).

Another possibility for grouping/classifying/clustering is to analyzethe content types of different messages and/or message segments. Somemethods for automatic (textual) annotation of image, video, sound, andother recordings, or for converting supplementary measured sensor valuesinto corresponding text information which, in combination, reflect thecontents of the messages, are available. The data resulting frommessages of different types of contents are able to be semanticallyanalyzed, processed according to predefined classification rules, andassigned via a linking matrix 390 which is configured as assignmentmatrix (n:m) for different classes/categories.

This also makes it possible to assign a message to a plurality ofclasses/categories. For example, an image message could be assigned bothto the class/category “portrait” (on the basis of the content), and tothe class/category “vacation” (on the basis of a date occurring duringvacation). The linkages then may be stored as result in an indexdatabase 300 configured as memory component having indexes of themessages of different classes/categories, and/or they are stored in amemory area for supplementary information of individual messages 52. Thememory area for supplementary information of individual messages 52 mayinclude, for example, the CLI and/or the user identification and/or theascertained age group and/or the gender and/or the determined emotionsand/or the acquired and/or ascertained supplementary information,determined identifications, Galileo data, temperature data, dataregarding air humidity, brightness and additional similar data.

In another development, the assignment of a message to a class may beprovided with weighting coefficients or confidence values which rate thereliability of the assignment to a particular class.

New possibilities for adapting and optimizing utilized technologies(i.e., FIG. 1) result from the large number of data obtained viaclassification and knowledge management and their linkings in integratedmessaging systems and/or from additional linkages whose data come fromexternal sources which were accessed via Internet/Intranet (21).

Furthermore, with the aid of a module for classifying and/or comparingwith available data of similar classes 400, the results of the languagedetection can be classified and compared to older detection results(text including possible N-best lists) of the same message class. Thisembodiment/method may be used to determine word sequences describing asimilar set of facts, for instance. By an additional application oftechnologies of associative knowledge management, implemented via amodule for associative knowledge management 410, new words/wordcombinations and/or phoneme sequences, for instance, may be found aswell which, when occurring in connection with specific message classes,are able to be linked to these classes and will then be available tofurther optimize the detection results within this message class. Thesewords/word combinations and/or phoneme sequences previously unknown tothe system are thus able to be utilized as key for additional linkages.

As described earlier, an image message may be assigned both to the“portrait” class/category and the “vacation” class/category. As aresult, the linkages are then stored in an index database 300 which isassigned to the mailbox system operator, and/or they are stored in thememory area for supplementary information of individual messages 52 ofthe mailbox owner. To support the additional application of associativeknowledge management, in an effort to simplify the search for messagesof specific categories, the sender information (user identificationssuch as sender ID, CLI, HLR. e-mail address, etc.) may be expanded bysupplementary information (such as date stamp, length, etc.) and linksthat point to the indexes assigned to the individual message classes(with a correspondingly similar annotation behavior) to which themessage was assigned as well. For instance, if a plurality of messagesof the “vacation2004” class has already been assigned for a sender,other messages whose content does not suggest this class at first glance(a portrait, for instance), also may be assigned to the “vacation2004”class via analysis of the time information and/or CLI.

In addition, it is possible to replace specific indefinable words/wordcombinations and/or phoneme sequences by words/word combinations fromsimilar utterances within the same message class and possibly by thesame speaker. A similar method for optimizing translation results,utilizing old data from the translation results of a message class thatcontain, for instance, pairs of word combinations/word sequences indifferent languages, may be used.

The entire messaging system and/or information from communicationnetwork 20 or from Internet/Intranet 21 and connected databases are/isthus utilized as background knowledge (world knowledge) to improve thecharacteristics of individual components such as that of ASR module 70with algorithms for language detection, translation system 190 and themodule for analyzing sender emotions 220.

Furthermore, it is an obvious thing to utilize data for locationdetermination, for example, via GPS, Galileo, and data regardingtemperature, humidity and brightness for the classification of messagesas well (see also FIG. 2). Some of these media types may also begenerated within the system, as supplementary information. For example,the brightness is able to be determined on the basis of the colorspectrum of attached instantaneous image information. Systems foracquiring various sensor data and their transmission via atelecommunication network are available. If these data are nowtransmitted together with messages left and received in a memory areafor supplementary information of individual messages 52, this willresult both in new classification possibilities within the messagingsystem and in more refined search criteria within individual messageclasses.

The operator may thus use the wealth of messages and their contents inthe messaging system to play the role of contact provider for theacquisition of contents. For example, if a mailbox owner is searchingfor “evening images of Naples”, this information may be used both todescribe a new class and, given available linkings with the classes“images”, “Naples”, to search for image messages that have acorresponding color spectrum and were sent at specific times of the day.All messages that include corresponding GPS data, for example, may beincluded in the class “Naples” as well. The creator of these messages,his or her consent having been obtained, may then be disclosed to thesearching mailbox owner.

The available classification rules may also be used to filter outunwanted messages (such as spam). Due to the structures introduced bythe present invention, this applies not only to pure text messages, butto the message types described in FIG. 2 as well.

As with parameters and parameter linkages included in the owner dataarea for acquired and/or ascertained owner data 170, the memory forstatic and dynamic parameter linkages for analyzing sender and ownerinformation 290, and the data memory with sample data 260, it ispossible to calculate, as classification result, a priority or a scoreof a message and/or its segments which is able to be utilized forsequencing or for acoustical and/or visual marking of the messagedocuments and/or their segments.

An acoustical marking may be implemented by, for example,

-   a) changing the voice, the volume and/or the prosody during    reproduction of texts via text-to-speech systems;-   b) omitting from the reproduction of text segments via    text-to-speech systems and/or from speech segments non-marked    segments or segments below or above a limit priority specified in    the memory for static and dynamic parameter linkages for analyzing    sender and owner information 290 or in the data memory with sample    data 260;-   c) signal tones; or-   d) a combination of the mentioned methods.

The illustrated exemplary structure for analyzing the message contentsis used to automatically generate responses to a message, possiblyaccording to parameter linkages specified in a mailbox-specific manner.If the mailbox owner is a public institution, for example, evenpreviously prepared document models may be sent to the particular senderfor completion upon evaluation of corresponding contents, or beforwarded to a language portal after the caller has left a voice mail,the portal offering the caller a variety of options (among them themailing of standard forms). If such a response has been generated by thesystem, the message left is amended by a report to this effect, so thatthe mailbox owner is notified of the automatic mailing of the answer.This may be done via a voice recording and/or a text message and/or viaother media (SMS, MMS, animation, image, smiley, etc.).

The system has an additional connection to a data network such asInternet/Intranet 21. This also allows a semantic analysis to beimplemented in order to evaluate external documents that provideadditional information regarding the available data in the system, forinstance the address data.

The parameter linkages stored in the data memory with sample data 260for analyzing sender information may be supplemented by additionallinkages that are used to generate automatic responses to alreadyclassified contents. This can be done both with a time offset by sendingan answer message to the sender, and in real time by a spoken dialogue,immediately upon leaving a voice mail. In this case, the asynchronouscommunication between at least two partners supported by a messagingsystem is converted into a synchronous communication. Standard responsesor parts of such for specific received message classes and/or theirsegments may be stored in various formats for this purpose, for example,in the memory for static and dynamic parameter linkages for analyzingsender and owner information 290. Responses automatically generated bythe system are linked or attached to the corresponding received messagedocument for the mailbox owner's information. Once the content of thememory for static and dynamic parameter linkages for analyzing senderand owner information 290 has been analyzed automatically, it is alsopossible to respond with an automatically generated order for goodsdesignations listed in the message document or its segments, forexample, via Internet/Intranet 21.

Additional automatically generated responses of the system followinganalysis of the content of received message documents and/or theirsegments may include:

-   a) generating and sending at least one message document in response;-   b) generating and sending at least one data file;-   c) establishing at least one telephone/voice connection to at least    one call line identification (such as call number, CLI, HLR, SIP    address, etc.) stored in the sender data area for acquired and/or    ascertained sender data 60 or in the owner data area for acquired    and/or ascertained owner data 170, or in the memory for static and    dynamic parameter linkages for analyzing sender and owner    information 290;-   d) generating and/or sending a signal;-   e) Combinations of the responses listed under a), b), c) or d).

For example, the system may already generate proposals for responses andoffer them to the mailbox owner.

1. A method for processing messages within the framework of anintegrated messaging system, wherein, after ascertaining theidentification, the text data and voice data to be gathered from theinstantaneous message of a sender are processed by a translation systemand, after processing, are analyzed together with sound data, imagedata, messages with other types of media in connection with theinstantaneous message, and data derived from additional informationsources; data that were stored within the framework of previous messagesof the sender in a sender data area for acquired and/or ascertainedsender data; and data of additional features from the owner data areafor acquired and/or ascertained owner data, such analysis beingimplemented according to semantic, prosodic, phonetic and otheranalytical methods, as well as image processing methods, for thepresence of information/data that are suitable to convey to therecipient both the message itself and also the background information inconnection with the particular message in the most authentic formpossible, both the instantaneous message and the data derived from themessage being evaluated according to predefined classification rules andassigned to different classes of a linking matrix configured asassignment matrix (n:m), and the data provided within the framework ofthe linking matrix being stored in an index database and/or in a memoryarea for supplementary information of individual messages.
 2. The methodas recited in claim 1, wherein, to translate the instantaneous message,the language of the message and/or a segment of the message isdetermined according to the relationL=argmax(W _(CLI)(Lx)+W _(LD) *C _(LD)(L _(x))+W _(T) *C _(T)(L _(x))+W_(H) *N _(H)(L _(x))+ . . . +W _(n) *C _(n)(L _(x))), the relation beingimplemented sequentially for each source language supported in thesystem, and the language (L_(x)) for which L exhibits the maximum valuebeing considered the source language (L) as a result, and in the case ofidentically high maximum values or maximum values for at least twolanguages that are very similar, information for correcting theweighting factors being output automatically.
 3. The method as recitedin claim 1, wherein the data record containing the result of thelanguage detection of the source language is analyzed both for thefurther adjustment with respect to the automatic selection of the sourcelanguage for the translation system and for the automatic selection ofthe target language into which the voice and/or text massage is to betranslated.
 4. The method as recited in claim 1, wherein the control ofthe individual translation modules of the translation module withdecision matrix is implemented via the decision matrix into which boththe result of the detection of the source language and the ascertainedtarget language are input automatically.
 5. The method as recited inclaim 1, wherein, in addition to the data resulting from the translationof the message, additional information is provided to the recipient ofthe message which is ascertained by utilizing principles of associateknowledge management.
 6. The method as recited in claim 1, wherein, inaddition to the data relating to the profile of the sender, theidentification such as name, first name, sender identification e-mailaddress, country code, gender, different call numbers and roleassignment for defined call numbers, the data record of the mailboxowner also stores data regarding source languages that are excluded froma translation.
 7. The method as recited in claim 1, wherein, toascertain the emotional state of the sender of a message, both theresults of the prosodic analyses of voice messages and the results ofthe semantic analysis of attached text files, as well as available imageand/or video information are being analyzed.
 8. The method as recited inclaim 1, wherein the gender and/or the age of the sender of the messageis ascertained with the aid of data from at least one module for speakerclassification and/or with the aid of data from a module forascertaining the most probable gender while including data from earliermessages from the sender data area for acquired and/or ascertainedsender data including supplementary information.
 9. The method asrecited in claim 1, wherein instantaneous data for the most probableemotional state of the sender of the message together with the dataregarding age and gender of the sender of the message are used forfurther statistical analysis; these data together with additional dataascertained by analyzing the sender information are combined into datarecords within the framework of parameters and/or parameter linkages, adivision being implemented into static and/or dynamic parameters and/orparameter linkages.
 10. The method as recited in claim 1, wherein thedata regarding the most probable emotional state of the sender of themessage is used to classify the messages in a priority list pertainingto the incoming messages.
 11. The method as recited in claim 1, whereinthe incoming messages are assigned to different message types, the dataresulting from the different message types being processed according topredefined classification rules and assigned to differentclasses/categories via a linking matrix, and the data, having beenassigned to the particular classes and/or categories, are assigned to anindex database of the mailbox system operator and/or stored incorresponding classes in the memory area for supplementary informationof individual messages of the mailbox of the individual mailbox owner.12. The method as recited in claim 1, wherein the data stored in theindex database of the mailbox system operator and/or the data stored inthe memory area for supplementary information of individual informationand sorted according to classes/categories are provided withcoefficients within the classes/categories, which determine thereliability of the assignment to a specific class.
 13. A system forprocessing messages within the framework of an integrated messagingsystem, comprising at least one translation system with automaticforeign language detection; at least one module for prosodic analysis ofspeaker emotions; at least one module for semantic analysis of senderemotions; at least one data memory with information regarding thelanguages that are able to be processed by the availableemotion-analysis module and/or the translation module(s); at least onedata memory with sample data and/or dynamic/static parameters and/orparameter linkages for analyzing collected sender information; at leastone sender data area for acquired and/or ascertained sender data towhich the data ascertained via the aforementioned module are supplied;and at least classification block.
 14. The system as recited in claimclaim 13, wherein the translation system is made up of a module forlanguage identification/language detection with the aid of a text,having access to an ASR module with algorithms for language detection; alanguage evaluation module connected to the module for languageidentification/language detection on the basis of a text, having accessto a memory area for supplementary information of individual messages,to a module for language detection, and to a module for ascertaining themost probable gender; a decision module connected to the module forlanguage identification/language detection on the basis of a text,having outputs to the input module for source languages, to an interfacemodule for selecting the source language and to a module forsupplementing the messages by supplementary information, the decisionmodule having access to the ASR module and being made up of atranslation module with decision matrix, which is linked to the inputmodule for source languages having the source languages (S1-Sn), theoutput module for target languages having the target languages (S1-Sm),the interface module for selecting the source language, and a module forselecting the target language, the module for selecting the targetlanguage having a cross connection to the owner data area for acquiredand/or ascertained owner data, and the output module for targetlanguages having cross connections to the module for supplementing themessages by supplementary information, to a module for analyzing senderemotions and to a classification block.
 15. The system as recited inclaim 13, wherein the classification block to which all data includingsupplementary data of a message are supplied according to the data type,is made up of a message classification module in which the messages aresorted according to their message type; a linking matrix, which isconnected to the message classification module and which is configuredas assignment matrix (n:m) for different classes/categories; and anindex database connected to the linking matrix, the messageclassification module having a cross connection to the memory for staticand dynamic parameter linkages for analyzing sender and ownerinformation, to the data memory with sample data and to the outputmodule for target languages, and the linking matrix and the indexdatabase having a cross connection to the mailbox.
 16. The system asrecited in claim 14, wherein the classification block to which all dataincluding supplementary data of a message are supplied according to thedata type, is made up of a message classification module in which themessages are sorted according to their message type; a linking matrix,which is connected to the message classification module and which isconfigured as assignment matrix (n:m) for different classes/categories;and an index database connected to the linking matrix, the messageclassification module having a cross connection to the memory for staticand dynamic parameter linkages for analyzing sender and ownerinformation, to the data memory with sample data and to the outputmodule for target languages, and the linking matrix and the indexdatabase having a cross connection to the mailbox.